Practical Machine Learning for Streaming Data with Python by Sayan Putatunda

Author:Sayan Putatunda
Language: eng
Format: epub
ISBN: 9781484268674
Publisher: Apress

Evaluation Methods

This section introduces model evaluation methods that are often used in evaluating the performance of supervised learning methods in a streaming data context. These methods are known as the holdout evaluation method and the prequential evaluation method (a.k.a., the interleaved test-then-train method).

In the holdout evaluation method, the performance evaluation happens after every batch of a certain number of examples or observations. The learner’s or model’s performance is evaluated on a test dataset formed by unseen examples. These unseen examples are used only for evaluation purposes and are not used to train the model. The same test dataset can be used in multiple evaluation runs. However, it is also possible to create dynamic test datasets. The holdout evaluation method can be applied using the EvaluateHoldout() function available in the scikit-multiflow framework’s evaluation module.

The prequential evaluation method, or the interleaved test-then-train method, is specifically designed for evaluating models in a streaming data context. Each of the examples is analyzed according to its order of arrival. In this evaluation method, the incoming unseen example in the data stream is first used for testing (i.e., the model/learner makes predictions). These unseen examples are also used to train the model before the next round of testing. The prequential evaluation method can be applied using the EvaluatePrequential() function in the scikit-multiflow framework’s evaluation module.

There is another evaluation method available in the scikit-multiflow framework (the prequential evaluation delayed method); however, this book sticks to the holdout evaluation method and the prequential evaluation method, with more emphasis on the latter.

This chapter uses these evaluation methods to track the performance of various supervised learning algorithms discussed in the next few sections. You need to specify the metrics when using either the EvaluateHoldout() function or the EvaluatePrequential() function.

In this chapter, classification problems generally use metrics such as prediction accuracy and the f1 score (the harmonic mean of precision and recall). For regression problems, metrics such as mean squared error and mean absolute error are used.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.